Staged Training Report ✓ Complete

Run ID: shoulder_session_multiheight_encoder_decoder_variable_mask_ratio
Generated: 2026-02-25 05:44:11
Stages Completed: 1
Total Elapsed Time: 07:02:36

Configuration

No config defaults changed since last commit.

All Staged Training Parameters (60 parameters)
ParameterValue
total_samples10000000
batch_size8
stage_samples_multiplier100000000000
update_interval250
window_size100
num_best_models_to_keep1
sampling_modeLoss-weighted
loss_weight_temperature0.5
loss_weight_refresh_interval50
stop_on_divergenceTrue
divergence_gap0.002
divergence_ratio1.5
divergence_patience50
divergence_min_updates10
val_spike_threshold2.0
val_spike_window15
val_spike_frequency0.75
val_plateau_patience250
val_plateau_min_delta0.0001
custom_lr0.0001
disable_lr_scalingTrue
custom_warmup-1
lr_min_ratio0.001
resume_warmup_ratio0.05
plateau_factor0.8
plateau_patience15
preserve_optimizerFalse
preserve_schedulerTrue
samples_modeTrain additional samples
num_random_obs_to_visualize2
selected_frame_offset3
runs_per_stage5
serial_runsTrue
clean_old_checkpointsTrue
enable_baselineFalse
baseline_runs_per_stage1
run_idshoulder_session_multiheight_encoder_decoder_variable_mask_ratio
enable_wandbTrue
wandb_projectdevelopmental-robot-movement
lr_sweep.lr_min1e-07
lr_sweep.lr_max0.01
lr_sweep.phase_a_num_candidates5
lr_sweep.phase_a_seeds1
lr_sweep.phase_a_time_budget_min3.0
lr_sweep.phase_a_survivor_count2
lr_sweep.phase_b_seeds3
lr_sweep.phase_b_time_budget_min10.0
lr_sweep.ranking_metricmedian_best_val
lr_sweep.min_samples_before_timeout1000
lr_sweep.min_evals_before_stop5
lr_sweep.save_sweep_stateTrue
plateau_sweep.enabledTrue
plateau_sweep.plateau_ema_alpha0.85
plateau_sweep.plateau_improvement_threshold0.0015
plateau_sweep.plateau_patience25
plateau_sweep.cooldown_updates5
plateau_sweep.max_sweeps_per_stage2
plateau_sweep.min_sweep_improvement0.0
initial_sweep_enabledTrue
stage_time_budget_min180
World Model Architecture (config.py)
ParameterValue
AUTOENCODER_LR0.0002
BATCH_SIZE1
CANVAS_HISTORY_SIZE3
DECODER_ONLY_DEPTH10
FOCAL_BETA5
FOCAL_LOSS_ALPHA0.1
FRAME_SIZE(224, 224)
GRADIO_UPDATE_INTERVAL1
LR_MIN_RATIO0.001
MODEL_TYPEencoder_decoder
PATCH_SIZE16
PERCEPTUAL_LOSS_WEIGHT0
SEPARATOR_WIDTH16
WARMUP_STEPS1000
WEIGHT_DECAY0.01
MASK_RATIO_MIN1
MASK_RATIO_MAX1
TRAIN_MASK_RATIO_MIN0.5
TRAIN_MASK_RATIO_MAX1.0

Timing Summary

Stage Plateau Sweeps Sweep Time Training Time Stage Total
Stage 1 7 01:48:08 00:15:55 02:04:03
TOTAL 7 01:48:08 00:15:55 02:04:03

Initial LR Sweep: Stage 1: selected LR 3.16e-05 in 00:14:51

Plateau Sweep Details

Total Sweeps: 7
Stages with Sweeps: 1 of 1
Total Sweep Time: 01:48:08
Average Sweep Duration: 00:15:26

Stage 1: 7 sweeps

LR Progression: 3.2e-05 → 3.2e-05 → 3.2e-05 → 3.2e-05 → 3.2e-05 → 3.2e-05 → 1.8e-06 → 1.8e-06

Sweep # Triggered At (samples) Wall Time Selected LR Duration
1 11,520 00:01:56 3.16e-05 00:15:11
2 22,528 00:18:57 3.16e-05 00:15:40
3 29,696 00:35:48 3.16e-05 00:15:37
4 44,544 00:53:53 3.16e-05 00:15:40
5 60,160 01:12:07 3.16e-05 00:15:18
6 78,592 01:30:28 1.78e-06 00:15:31
7 87,808 01:47:32 1.78e-06 00:15:09

Stage Results

Stage Best Loss Stop Reason Samples Trained Time Sweeps LR (Initial→Final)
Stage 1 0.036345 max_sweeps (2) 7,680 02:04:03 7 3.2e-05→1.8e-06

Total Plateau Sweeps: 7

Stop Reason Breakdown

Loss Across Full Training Run

Loss Detail (Post Initial Drop)

Multi-Run Statistics

Total Runs: 5
Average Best Loss: 0.071393 ± 0.037210
Best Overall: 0.036345
Worst Overall: 0.118392

Stage 1 (5 runs)

Run Best Loss Stop Reason Samples Time Selected
1 0.042132 max_sweeps (2) 14,848 01:44:48
2 0.118392 max_sweeps (2) 7,936 00:35:28
3 0.115258 max_sweeps (2) 9,984 00:36:19
4 0.044837 max_sweeps (2) 11,520 01:44:39
5 0.036345 max_sweeps (2) 7,680 02:04:03
Mean: 0.071393 ± 0.037210 Min: 0.036345 / Max: 0.118392 Range: 0.082047

Best Checkpoint

Name: best_model_auto_session_so101_multiheight_part1_1345_shoulder_session_multiheight_encoder_decoder_variable_mask_ratio_00089088_cont_val_0.036345.pth
Stage: 1
Hybrid Loss (full session): 0.044051

Learning Rate Timeline with Plateau Sweeps

Stage Progression

Stage Orig Loss Train Loss Time Samples Stop Reason
1 ⭐ 0.044051 0.036345 02:04:03 7680 max_sweeps (2)

Hybrid Loss Over Original Session (per Stage)

Stage 1 (Best) - Hybrid Loss: 0.044051

Sample Counts

Cumulative Across All Stages

Per Stage

Stage 1 (Best) - Total Samples: 7,680

Best Checkpoint Inference

Selected Frame 3

Action 0

Action 1

Action 2

Random Observations

Observation 1316

Action 0
Action 1
Action 2

Observation 156

Action 0
Action 1
Action 2